1. Technical Field
The invention is directed to a method of displaying virtual information in a view of a real environment in which a system for displaying of virtual information in a view of a real environment is provided and a current pose of at least one part of the system relative to at least one part of the real environment is determined.
2. Background Information
In the prior art, applications are known which augment an image or images generated by at least one camera with virtual information or objects using the so called Augmented Reality (AR) technology. In such application, a camera coupled to a processing unit such as a microprocessor takes a picture of a real environment, wherein the real environment is displayed in a view to the user on a display screen and virtual objects may be displayed in addition to the real environment, so that the real environment displayed on the display screen is augmented with virtual objects of any kind. As a display, a display screen (e.g. of a mobile device, such as a smart phone) or an optical see-through display device (e.g. a semi-transparent head mounted display) may be used. In such application, in order to augment the image with virtual objects, there is the need for the microprocessor to determine the current position and orientation (termed in the art as “pose”) of the camera with respect to at least one object of the real environment in order for the microprocessor to correctly augment the captured image with any virtual objects. In this context, correctly augmenting the captured image with any virtual objects means that the virtual objects are displayed in a manner that the virtual objects fit into the scene of the image.
In the past, various publications about Augmented Reality and Virtual Reality technologies are available.
The present application is concerned with solving a detailed aspect of creating an improved method of displaying virtual, location based information. Particularly, in Augmented Reality many concepts of displaying virtual information in a view of a real environment have been made public. In Virtual Reality the concept of “level of detail” is known. Here, different representations of virtual objects can be displayed according to graphic-power of the displaying system or distance to the object.
The present invention solves a problem which has become eminent only recently, with mobile devices and location based services becoming possible and widespread. Location based services, especially the ones overlaying virtual data on top of the view of the real world (e.g. a camera image or through a semi-transparent display or by projective means, projection information on the surrounding environment), often rely on different means of localization (optional) and means of getting the orientation (optional). In other words, location based services using AR technologies often use different pose determination means. The means can use data from sensors, which are moving with the user/device (inside-out-mechanisms) or sensors, which are measuring the users' movement from outside (outside-in-mechanisms).
As a simplification, it is referred to in the following and throughout the whole application to a “pose”, which could be the position in 1, 2 or 3 degrees of freedom or position and orientation in different degrees of freedom or just orientation. The pose can be relative to an earth-centered coordinate system or any other coordinate system (e.g. of a large building). The virtual information is directly or indirectly positioned with respect to the coordinate system.
Many methods have been described in the state of the art to determine the pose of an information system in respect of a coordinate system. Some of the methods use, for example, GPS, compass, or gravity-sensors, other methods rely on images or distance-measurement-devices, and many other methods are also known. Often these methods combine different sensors to estimate the pose, or switch between different sensor-systems, depending on their availability. Sometimes, also the whole method is switched from one type of pose determination method to another type of pose determination method.
Further, different sensors and/or different pose determination methods have different accuracies in determining the pose. For example, a pose determination method based on a GPS sensor operates at a lower level of accuracy (or higher level of inaccuracy, e.g. operates with a higher uncertainty) as compared to an optical based tracking method for determining the pose which operates at a higher level of accuracy (or lower level of inaccuracy, e.g. operates with a lower uncertainty). In combination with various kinds of real world scenarios, this may result in problems when displaying virtual information in a view of a real world.
For example, there may be scenarios in which virtual information is to be displayed at a rather high level of accuracy in order to fit into the scene, for example when the level of detail of the real world is rather high at the location where the virtual information is to be displayed (e.g., in an area of the real world which is rather close to the user). In such a case, the aspect of pose accuracy is rather important. On the other hand, there may also be scenarios in which the level of detail of the real world is rather low at the location where the virtual information is to be displayed (e.g., in an area of the real world which is rather distant to the user) and where the aspect of pose accuracy is less important.
Therefore, it would be beneficial to provide a method of displaying virtual information in a view of a real environment which takes account of the above mentioned aspects.